On the Statistical Efficiency of the LMS Family of Adaptive Algorithms

AbslrabTwo gradient descent adaptive algorithms are compared, the LMS algorithm and the LMSNewton algorithm. LMS is simple and practical, and is used in many applications worldwide. LMWewton is based on Newton's method and the LMS algorithm. LMSiNewton is optimal in the least squares sense. It maximizes the quality of its adaptive solution while minimizing the use of training dah. No other linear least squares algorithm can give better performance. LMS is easily implemented, but LMWewton, although of great mathematical interesr cannot be implemented in most practical applications. Because of its optimality, LMWewton serves as a benchmark for all least squares adaptive algorithms. The performances of LMS and LMSmewton are compared, and it is found that under many circumstances, both algo. rithms provide equal performance. For example, when both algorithms are tested with statistically nonstationary input signals, their average per. formances are equal. When adapting with shtionary input signals and with random initial conditions, their respective learning times are an average equal. However, under worst.case initial conditions, the learning time of LMS can be much greater than that of LMSmewton, and this is the principal disadvantage of the LMS algorithm. But the strong points of LMS are ease of implementation and optimal performance under important practical conditions. For these reasons, the LMS algorithm has enjoyed very widespread application.